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ABSTRACT 

How various methodological decisions may influence 
studies of ^he effect of time-on-tastc on achievement are examined, 
Subiects were students in grades 2*5 in 18 classes taught by 12 
^•eachers in a rural Barvland school district. All students vere 
pre-tested in Februarv 19^8 iii reading, language arts^ math and 
social studies using the Comprehensive Test of Basic Skills, A 
post^test was given in *!av# 1978* It was found tliat altering 
definitions of time-on-task to include momentary off*task behaviors 
affected the conclusions for Ihe importance of time*on*task. Clear 
evidence was presented that sampling segments of instruction vould 
tend to obscure the positive results for time-on-task. It nas also 
shown that re^acina the number of days of observation weakened the 
effects of time-on-task* However* the timina of the observation was 
not very i..portant for the noted effects. The effect of sampling 
fewer than six students was explored and, due to the effect on 
reliability, it was sugges^-ed that this approach vould not be 
advisable, Pesults suggested that although there is an understandable 
urae to lessen the observation time in order to bolster tha number of 
set^inas observed* such steps should only be taken cautiously, 
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Introdi^ctor> Siratement 

Tije Center for Social Organization of Schools has two primary objectives; 
to develop a scientific knowledge of how schools affect their students, and 
to use this knowledge to develop better school practices and organization. 

The Center works through four programs to achieve its objectives. 
The Studies in School Desegrej^atlon program applies the basic theories 
of social organization of schools to study the internal conditions of deseg- 
regated scliools, the feasibility of alternative desegregation policies, and 
the interrelation of school desegregation with other equity issues such as 
housing and job desegregation. The School Organization program is currently 
concerned with authority-control structures, task structures, reward systems, 
and peer group processes in schools. It has produced a large-scale study 
of the effects of open schools, has developed Student Team Learning Instruc- 
tional processes for teaching various subjects in elementary and secondary 
schools, and has produced a computerized system for schoolwide attendance 
monitoring* T^he School Process and Career Development program Is studying 
transitions from high school to post secondary institutions and the role of 
schooling in the development of career plans and the actualization of labor 
market outcomes. The Studies in Delinquency and School Environments program 
Is examining the ln^:eraction of scJ-'Ool environments, school experiences, 
and individual characteristics in relation to in-school and later-life 
delinquenc> . 

This report, prepared by the School Organization program, excmines 
the methodological problems Involved In studies of time-on-task in classrooms. 



Abstract 

Many recent studies have reported benefical effects of time-on-task 
on student academic acliievement • This paper examines the methoaological 
problems involved in measuring t lme-on-task> especially problems related 
to the definition of off-task behavior, length of observation times, 
days of ohservations, scheduling of observations, and sampling of students 
for observation* The findings show that the methodology selected can 
influence the results of time-on-task studies 
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Introduction 

Research interest has recently focused on the centrality of tlme-on-task 
for understanding classroom effects and effectiveness (Fisher ct al.> I976a> 
1976b; Filby and Marliave, 1977; McDonald and Elias^ 1976; Cooley and Lein- 
hardt, 1978), This research has provided important evidence that links 
cl^issroom practices, rime-on-task and learning outcomes. Although the evi- 
dence in general points to positive and meaningful effects of time-on-task^ 
the results are not consistent across studies nor across grade levels/ 
subject matters within studies (e,g, the results obtained in the Beginning 
Tea<^her Evaluation Studies, BTES , for mathematics/reading at grades 2/5), 
Moreover, the effects documented for time-on-task, although positive^ have 
not been uniformly l^irge.^ Nonetheless, the effects for time factors have 
assumed appreciable stature by virtue of the fact that time factors can be 
altered, whereas more statistically Important factors, such as family back- 
ground or entering aptitude, are difficult or even impo sslble to alter* 

Thus^ the use of time in classroom continues to be a central theme In 
educational research. The fact that the results are modest ana inconsistent 
has been attributed to particular methodological or research design problems^ 
not problems with the assumptions guiding the research. That is, the assump- 
tion that classroom practices have appreciable impacts on time«on-task 
which in turn affect the degree of learning is generally not at issue. The 
present state of encouraging but not entirely clear results is taken to 
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indicate the exlf^tence of methodological as opposed to theoretical problems. 

Given this slant on the problem^ ic seems reasonable to ask to wliat 
extent the nature of the present findings are due to pcttticular methodolo- 
gical choices or decisions. In particular, it «eems useful to explore ru^w 



the observation scheme used, the timing of the observation, the length of 
the observation and the number cf observations may affect the det^xtion of 
tlme-on-task effects. The present paper uses an existing set of observa- 
tional data and manipulates it to conform to alternative sampling, timing 
and definitional choices. Using these alternative choices* we then compare 
the effects obtained for time"on-task with results reported previously 
(Karweit and Slavin, 1980), We examine alternative choices in 5 areas: 

Ip definition of off-task behavior 

2p length of observation visit 

3. days of observation 

Ap scheduling of observation 

5p sampling of students for observation 

Data 

The data were collected in four elementary schools in a rural Maryland 
school district. All schools had open space construction but used essen- 
tially traditional methods of classroom organization and instructionp Sub- 
jects were students in grades 2"5 in 18 classes taught by 12 teacherSp All 
students were pre- and poiit-tested in February 19V8 in reading, language 
artSj !?.ath and social studies using the Comprehensive Test of Basic Skills* 
Stucients in each class were assigned to the top third, middle third, or lower 
third of the class based on the pre-test information, and two students (one 
boy and one girl) were chosen from each third for observationp The obser- 
vations were thiif. conducted for six students per class, 108 total students, 
through tiie ^^econd semester of 1978, and the post-test was given in May, 
1978. 

Students were observed during their mathematics classes, which averaged 
50 minutesp Each classroom was observed for at least nine daySj and some for 
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as many as eighteen days. The observers recorded three pieces of Informa^ 
tion for all six students during a thirty second Interval; the nature of 
the task (procedural^ seatwork^ or lecture); the student's response to the 
task (on^taskj off^task or no .task opportunity), and the content of the In- 
struction (e.g., two digit multiplication, or going over p. 1A7) , 

All six students were observed in a predetermined order every trhirty 
seconds. To determine on^ or off-task behavior, the observer took a quick 
look at the student's behavior and recorded the response at that particular 
instant. The observers were trained not to dwell on deciding whether n be- 
havior was on- or off-task, but to record their first impression in accor~ 
dance with established definitions of on- and off-task responses. 

On average, 100 observations per day were recorded for each student, 
detailing the task, the content of instruction, and the response. Across 
all days of observation, we logged about lOOO observation points for each 
student in the sample, or about 110,000 observation points. 

Because of the size of the data base, we entered the task, content 
and response codes in a summary form which maintained the essentials of the 
information. Each entry pertained to a specific task or activity and gave 
the number of seconds each child was on-or off-task during that time. For 
example^ if the class were involved in se itwork during the first ten minutes 
of the class and then the teacher explained the seatwork during the next 
eight minutes, crented two entries, one detailing the on/off task behavior 
du>"inji s<>c3twork, and the other giving tho on/off task behavior for oacii of 
the six chiJfiren ciurlng tfic te.icficr^dlrectod ;ictlvlty. Prom tliese data* a 
'May'* record w^is constructed which sutnmnrized the d;itly task, content and 
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response patterns for each child. 

In adcliticn, a special data set containing each 30 second record of 
task^ content and response was compiled for five o£ the eighteen classrooms* 
These supplemental data will be used along with the basic data in tlie ana- 
lyses . 

Definition of On- and Off-Task 

On- and off-task behaviors were coded during instructional portions 
of the lesson only. However^ a ciiild could alKO have a response other than 
on- or off-task during instructional time. The diagram below depicts the 
different categories and when they could occur in this observational scheme. 



Procedural Tima 



Allocated Time 



Instructional Tire 



other 




Off- 


On- 


response 




task 


cnsk 



The allocated time was the clock time spent for the mathematics class. 
Procedural time was any time spent lining up, receiving instructions, being 
involved in disciplinary action, going to fire drills^ being interrupted 
by the P- A. system and the like. Instruction pertained to th^ time spent 
specifically on mathematics instruction; discussion of world events^ elec" 
tions> snow storms and other material not pertaining to math was ^ot coded 
as instructional time. On-cask behavior was defined as behavior appropriate 
CO the tnsk at hand. The definition of appropriate behavior depended upon 
the ta^^k ami specific rules of the classroom, "Other response" was used to 
Cover sUijations fii wlilofj tlie child ums not OTl-t^^^^k but was not off-task 
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eicheri Such situations arose when the child was sharpening a pencll> 
walking to another part of the room Co obtain new materials, waiting for the 
teacher to he]p with a problem, or doing some other activity because the 
original assignment was finished. 

We focused on two particular problems in assessing off-task behavior. 
One, involved the effect of including momentary off-task behavior; the other 
involved tlie effect of including no-tas'k-opportunity time (i.e* "other 
response") as off-task behavior, 

a. Momentary off-task behavior 

During any class period, children may momentarily gaze out the window, 
fidget, or otherwise be momentarily distracted. On the one hand, this 
momentary off-task time can be looked upon as insignificant for the learning 
process. On tlie other hand, momentary off^task behavior may be signalling 
declining attention and motivation and might therefore be important for 
understanding; the learning process* In the BTES analyses, o[f-task behaviors 
shorter than one minute wrre not counted; i*e* these flickers of inattention 
were not considered consequential. In coding the data used in the present 
study, we included all off-task behavior > regardless of duration. To assess 
wlietlier tills decision to include short-term inattention affected the results 
obtained, we changed all off-task behavior of less tlian one minute to on- 
task and repeated the analyses for the supplemental sample of five class- 
rooms. The average r<ite of on-task behavior increased from .79 to *83 
and tlie standard deviation was reduced from .08 to .07, Including the 
momentary off-task beltavior yielded cotrelatkons of between on-task 
and pre-test score and .45 with post-test sic* * Excluding momentaiV off" 
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task behavior, these correlations became *33 and *39* We carried out re- 
gressions of post-test on pre-test and the alternative measures of on-task 
behavior. Using the measure which excluded momentary off-task behaviors 
produced more modest results (p ^ *10) than did using the more Inclusive 
measure (p ^ .05) . 

Whether momentary inattention is included or not should be based on 
the particular model of learning one has formulated* Certain views o£ Ihe 
learning process may be compatible with inclusion of these momentary distrac- 
tions; other views would not be* The present exercise was not intended 
to shed light cn whether a particular point of view is proper or improper^ 
but to illustrate that the methodological decision to include/exclude these 
flickers of inattention affected the results obtained. 

b. Other response and off-task behavior 

The dichotomy of on- or off-task provides a working categorization of 
student responses to instruction, but there are numerous ambiguous situa-- 
tions in which the student is not on-task^ yet could not be considered off- 
task. For example^ a student may have finished an assignment and have 
nothing more to do. Students who finish early are likely to be those who 
need less tlme^ i.e.^ have more aptitude for the particular task at hand; 
thus the amount of finished time should be positively related to achlevement> 
In contrast to the negative relationship of off-task variables and achieve- 
ment. In our data> the correlation between finished tim** and post-test 
score ^jas .19 while the correlation between off^task and ^jost-test score 
was -.28. Tn rGp;ressions (not detailed here) in which finished time was in- 
cluded with off^task time> the effects jf off-t>^sk time were dimlnislied 
v^ppreciably. 
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Length ot observation visit 

An important design consideration is the length of the observation 
period* One could observe a single classroom all day long, for some fixed 
fraction of the day> or for some specific instructional program, Or> a 
combination of these lengths of observation might be used. Because our 
interest was in how the use of time affects mathematics achievement, we ob-- 
served students during their entire mathematics instruction* It was nor 
possible (given our budget constraints on observer time) to observe all 
teachers within a school* An alternative decision wr^uld have been to ob- 
serve more teachers, but for some smaller segment of their mathematics in- 
struction* We might have decided, for example, that instead of visiting 
one teacher for sixty minutes we might have used one of the combinations 
below: 

NO* TEACHERS NO* MINUTES TOTAL TIME 

2 30 60 

3 20 60 
6 10 60 

The choice among these alternatives is basically between getting enough 
classrooms to provide stable estimates of the effect of time-o'^-task, and 
scheduling sufficient timt to ensure that the observed behavior is represen- 
tative. If time-on-task is distributed fairly uniformly across the day or 
the period of instruction, then a time sample may be entirely adec^uate* 
Table 1 f^ives the means and standard deviations of time-on-task for nine 
clays of observation In one classroom* The first columns provide statistics 



Table 1 About Here 
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Table 1 

On-Task Rate for Selected Portions; 
of Mathematics Instruction 
in one classroom 



Minutes Minutes Minutes 

1-10 1-20 1-30 



Day 


X 




X 




X 




1 


.906 


.066 


.878 


.046 


.865 


.051 


2 


.911 


.086 


.815 


.059 


.805 


.092 


3 


.922 


.078 


.923 


.079 


.921 


,094 




.739 


.236 


.8.18 


.062 


.775 


.062 


5 


.817 


.002 


.690 


.158 


.653 


.195 


6 


.889 


,087 


,869 


,073 


.804 


.099 


7 


.884 


.169 


.866 


.188 


.809 


.175 


8 


.958 


.066 


.928 


.063 


,889 


.088 


9 


.825 


.196 


,841 


.148 


.E.50 


.171 


x" 




872 




.848 




819 


X this 


seRnienC 






,824 




.761 
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for the first 10 minutes of class; the second columns for the first 20 
minutes^ and the third for the first 30 minutes, Tho overall mean for the 
time period is supplied as well as the mean £ot the particular 10 minute 
segment. The average on-task time in this class was markedly higner during 
the first 10 minutes of instruction than it was for the next 10 (^r 20 minutes. 
Clearly^ the timing of observation in this classroom was important ic: the 
results obtained as tlme^on-task was not distributed evenly across the 
mathematics class time. Other classrooms exhibited different patterns of 
high and low attention* Some classes started off with lower on-task rates^ 
seemed to warm up to Instruction^ and have higher on-task rates, and tlien 
to die down* Still other classrooms had no consistent pattern at all. Con- 
sequently^ It is difficult to predict v*hat the effect in general would be 
if selected portions only of the class time were observed* Thus> although 
the effect of observing shorter periods may not be consequential for the relia- 
bilities obtained (see Rowley^ 1976), how those periods are selected may be 
very consequential. 

to illusti this pointy we regressed post-test achievement scores on 
pre-test scores and alternate measures of on-task rate> namely measures from 
the first ten, twenty, thirty and fifty minutes of instruction* The F values 
obtained for the time-on-task measures were <010^ 1.22> 3.0^ and 4»34, respec- 
tlvely< The n of this sample was extremely small (22 students): however^ 

the results suggest that observing for shorter segments would have appreciably 

3 

altered the effects obtained* 

Altering the number of days ot observation 

Conventional wisdom has it that about ten days of observational data 
should be sufficient to accurately portray the activities of a classroom. 
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However , few studies have investigated the effects of observing classrooms 
for fewer or more days, aven though this question Is of considerable design 
and practical -importance. If we can obtain sufficient information in a 
shorter period (e.g. five days irstead of ten), it would be possible to 
observe substantially more classrooms without appreciably altering the obser- 
vation costs. 

Tn the present data set, we observed some classrooms for as many as 18 
school days and others for as few as 9 days* With these data, then, we can 
pretend that we had observed a fixed number of days (e.g* 3, 4, 5, 6, 7, 8, 
9) and assess how this observatiori schedule would have affected the detection 
of effects of time-on-task on achievement. We think of time-on-task as a 
variable which is influenced not only by an individual child's disposition, 
aptitude, and idiosync acies, but also by the instructional setting in the 
class and by external event^s such as the daily weather. Each child may have 
a stable tate of on-ta&k behavior with daily fluctuations depending upon 
his response to the classroom and other environmental settings* Given this 
view of time-on^-task as a variable, a natural way to capture the daily 
and individual variatloii is to view each day*s time-'on-task as an item In 
a scale of total time-on-task. We can then see how consistent the behavior 
is across a differiitg number of days or items in the scale* 

As expected, increasing the number of days does provide an overall in- 
crease in reliability. The median coefficient alphas obtained for 3-9 
ciays were *54, ,57* *7X, .73, .79, .81, .82* Whether the increase in relia- 
bility obtained from observing nine days vs* 5 days Is consequential de* 
pends on the effecr one is trying to document. Hecciufse relic, Ulty deter- 
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niines the maximal correlation that cnft can find between achievement out- 
comes and farte-on-task, the obtained reliability is of some consequence* 
To assess the effect of these variations in reliability, we used the third 
grade sample (n'=36) and regressed post-test CxBS score on pre-test and 
alternate measures of time-on-task, namely measures obtained from: 

1* five days observation 

2* nine days observation 

3. eighteen days observation 

Table 2 shows that had we observed tor the first five or first nine 
days Our effects for time-on^task would have been much more modest* The 

Table 2 About Here 

"18-day" results ^uow significant effects for on-tssk minutes, engagement 
rate and off-task rate* Had we observed the same classrooms, but for fewer 
days, the results obtained would have been much weaker* 

It is possible that the days at the end of the observation were signi^ 
ficantly different from the ddys at the beginning; if this were the case 
we would be witnessing an effect for timing and not for length* This issue 
is explored in tl>e next section* 
Tiniing of o bservation days 

ThrougTitni^ the school year, there are no doubt more intensive and less 
Intensive tMes for classroom instruction* For example, one obviously would 
not want to schedule observations of attending behavior in the few days 
prior to the Christmas or summer break* Besides these obvious cyclical 
differences in tlTne-on-task, there may be less obvious sources such as 
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Table 2 

Comparlsc:i of Results Obtained 
for tlme-on-task using 5, 9, and 18 
days of observation 



time-on -task 
rate 

time-on-task 
minutes 

time-off-task 
rate 

time-of f-task 
minutes 



18 days 
b/beta F 



9 days 
b/beta F 



4V.51 



».56 



(.178) p^.05 

.249 5.U 

(.165) p<.05 

-48,1 A. 33 

-(-147) p<,05 

-.450 2,86 

-(-121) n.s. 



33.62 3.21 

(.129) p<.10 

.156 2,28 

(.111) n.s. 

-.32.9 2.07 

'(,10) n.s. 

-.329 1,39 

■(■09) n.s. 



5 days 
b/beta F 



32.25 3,62 

(.138) p 4. .10 

.131 2.19 

(,110) n.s. 

-.109 .141 

-(,03) n.s. 

■16.76 .459 

■(.05) n.s. 
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different teache'' expectations for tlme-on-task depending upon the time of 
the year and coverage of material by that point. For example > ^^e might hypo- 
thetlcally view the school cal^indar from the perspective of the intensity 
of teacher effort as followf^: 



HIGH 



intensity of 
teacher effort 



LOW 




Sept Oct Nov Dec Jan Feb Mar Apr May June 



r 

Here, January, February /^and March are more serious months because of 
the on-coming deadline of the erid of the school year and because there Is 
3:111 time available to redrfess learning deficiencies. 

We are able in a limited fashion to see if time-on-task differs by 
time of year> using these data. For five classrooms we observed students 
for a nine-day period in February and also In May. The means and standard 
deviations for these classrooms are provided In Table 3 for the two different 
time periods* Table 3 also provides the reliabilities; for the two periods 
of observation (column 5 and 6) and for two mixed scales (SI and S2) composed 



Table 3 About Here 



of equal number of items from February and May. The reliabilities and the 
means do not appear to be very different for the two tlmi^ points. This 
table supplies limited evidence of the consistency of the classroom over 
time, which suggests that the timing of the observational period may not 
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Table 3 

Comparison of Mean values and reliabilities obtained 
for time on task in February and May 



Feb. 
Means 


May 
Means 


Feb. 


May 


Comb ined 


SI 
o< 


S2 


.8U 


.856 


.92 


.96 


.97 


.93 


.94 


.899 


.900 


.76 


.42 


.71 


.70 


.53 


.929 


.930 


.67 


.76 


.85 


.70 


.70 




.847 


.85 


.76 


.79 


.56 


.71 
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be all that consequential* It also suggests that o^jt rallure to find 
significant effects for time-on^task ustng only nine days of observational 
data was most likely due to the decreased reliability of the scale and not 
due to scheduling effects,. 

Altering the number of students sampled in the classroom 

Another decision which has to be made is whether to observe all stu- 
dents in the room or to follow a sample of students* Whether to observe 
the entire classroom or selected students depends largely on the purpose of 
the observation* If one is interested primarily In how classroom organi- 
zation affects time-on'-task, the entire class would probably be observed* 
Other strictly pragmatic elements such as high absenteeism or sensitivity 
of identifying students for observation may influence this decision* 

Given that the practical and theoretical concerns dictate that sampling 
should take place, the question is how many students are needed to obtain 
a reliable estimate of the on-task behavior for the class* We can examine 
this issue in two ways with these data* In one classroom, we actually ob- 
served twelve students as opposed to six, and comparing the class meaas and 
standard deviations and reliability obtained for these six vs* twelve shows 

them to be very similar (x = *87, x = *86, r - *92, r = *89) * Another 

12 ^ 12 6 

way we can focus on this issue is by reducing the number of studeats and com- 
paring the obtained reliabilities* We used a random selection of three of 
the six students to assess the effect this sampling might have on reliability* 
Tiie median r(?l iablUtlef; wert^ not appreciably reduced l)y sr?Jcct1nK onJV thrvc 
students.^ Hovjever^ gWea the fragility of time-on-tosk effects which we 
have documented here, it would seem worthwhile to keep reliability as hii\h 
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as possible. In this Instance, observing six students would seem desirable* 
_Suinmary an_d discussion 

This paper has examined how various methodological decisions may influ- 
ence studies of the effect of titne-on-task on achievement* We found that 
altering definitions of time--on-task to include moiaentary off-task behaviors 
affected the conclusions for the importance of time-on-task*- We found 
clear evidence that sampling segiaents of instruction would tend to obscure 
the oofiltlve results for time-on-task* We further showed that reducing 
the number of day5> of observation also weakened the effects of time*on-task* 
The timing of the observation was not very important for the noted effects^ 
however* Finally> we briefly explored the effect of sampling fewer than six 
students and, due to the effect on reliability, suggestea that this approach 
would not be advisable. 

The findings in this paper suggest that although there is an understan- 
dable urge to lessen the observation time in order to bolster the number of 
settings observed, such steps should only be taken cautiously* Wliether 
the effects detected and not detected here are bound up with the particulars 
of this observation fjtudy can only be determined by more systematic examin- 
ation of these methodological Issues. In thi5 5ense, we hope the paper 
serves more as a source of what the question might be than of what the answer 
is* What this paper does show is that methodological decisions, including 
some that appear quite minor^ can have major consequences for the conclusions 
that are drawn from observational data* 



9p 



-17- 
Notes 

1* A typical finding has been chat tlme-on^task when added to a regres- 

sion o£ post-test on pre-test will increment R by about 3 percent* 

2 

Although increments to R provide a conservative view of the imp^ortance 
of a variable, other indicators, such as the magnitude of the beta 
weight or the residual variance accounted for, have not been substantial 
either * 

2, An alternative perspective would be that the work is basically atheo- 
retical so that it is natural £a/fault the methodology* 

3* For five of the eighteen classrooms, we coded each 30 second interval 
of task, content and response* From this sample, the twenty-two stu-^ 
dents who had complete test and observational data were used in the - 
regressions reported in this section, 

A* The median reliabilities obtained for three students in comparison 
to six students for three to nine days of observation are: 

3 days A days 5 days 6 days 7 days 8 days 9 days 

3 

Students *A3 .65 *63 - *63 .71 * 77 *8l 
6 

Students .5^ .57 . 71 .73 .79 .81 .82 
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